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We describe the present status of the computing system in the Belle experiment at the KEKB e+e~ asymmetric- 
energy collider. So far, we have logged more than 160 fb~^ of data, corresponding to the world's largest data 
sample of 170M BB pairs at the T(45') energy region. A large amount of event data has to be processed to 
produce an analysis event sample in a timely fashion. In addition, Monte Carlo events have to be created to 
control systematic errors accurately. This requires stable and efficient usage of computing resources. Here we 
review our computing model and then describe how we efficiently proceed DST/MC productions in our system. 



1. Introduction 

The Belle experiment T is the S-factory 
project at KEK to study CP violation in B me- 
son system. The KEKB accelerator f5j is an asym- 
metric energy collider with 8 GeV electron to 3.5 
GeV positron, operating at T{AS) energies. The 
Belle group has been taking data since June 1999 
and has logged more than 160 fb^^ of data un- 
til December 2003. This means that we have the 
largest data sample of B meson pairs around the 
T(4iS') energy region in the world. The KEKB 
reached its design luminosity of lO'^'' cm~^sec^^ 
on May 2003. The KEKB performance is still be- 
ing improved and we can accumulate integrated 
luminosity of about 800 pb ~^ per day recently. 

We have to promptly process all of beam data 
to provide them for user analyses. To do this, the 
DST production should be stable and the com- 
puting resources have to be used in an efficient 
way. The Monte Carlo ( MC ) data should be 
generated with statistics large enough to control 
experimental systematics. As a result, data size 
which we should handle is extremely huge and 
a mass storage system has to be used to avoid 
network traffic, and data management for entire 
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data sets should be carefully done without loosing 
flexibility, for instance, any modification of data 
distributions. 

Provided a large amount of data sample, we 
have published a variety of physics results re- 
lated to B meson decays, which is highlighted by 
the first observation of the large CP violation in 
B meson decays 0. The quick and stable data 
processing greatly contributed to this remarkable 
achievement. 

In this paper, we will describe a detail of our 
computing system after a brief sketch of Belle 
software utilities. In the next section, how we 
proceed DST as well as MC productions will be 
mentioned, and then summary will be given. 

2. Belle Softv^rare 

2.1. Core Utility 

In the Belle experiment, unique software frame- 
work called as B.A.S.F.( Belle AnalySis Frame- 
work ) for all phases in event processing from on- 
line data-taking to offiine user analyses has been 
employed. This is "home-made" core software de- 
veloped by the Belle group. In this scheme, each 
program, written in C-|--f , is compiled as a shared 
object, and it is treated as a module. When one 
wants to run a program with this framework, the 
modules defined in the one's script are dynami- 
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cally loaded. 

The data handling is done with the traditional 
bank system, named as PANTHER, with a zlib 
compression capability In this utility, data trans- 
fer between different modules is made and data 
I/O is maniplated. PANTHER is only software to 
handle our data in any stage of data processing. 

The typical event data size for rawdata is 35 
KB, and it is increased up to 60 KB for re- 
constructed DST data, which contains all of de- 
tector information after unpacking, calibration 
and analysis. For user analyses, compact data 
set ("mini-DST") is produced, which is approxi- 
mately 12 KB for one hadronic event. 

2.2. Reconstruction and Simulation Li- 
brary 

The reconstruction package is built when major 
update of programs, which can affect final physics 
analysis, is made. Usually it is built once or twice 
per year. Once new library is released, we need to 
process all of events to produce a consistent data 
set for analysis. 

For MC data, the detector response is simu- 
lated based upon the GEANT3 library (J. Here 
background events, calculated from beam data, 
are overlaied onto MC events. The same recon- 
struction codes are also applied to MC simulated 
events. 

The detector calibration constants are stored in 
the database, for which PostgreSQL[S] is adopted 
in the Belle experiment. Two database servers are 
running in our computing system, where one is for 
public usage and the other for DST/MC produc- 
tions. The contents of each server are periodi- 
cally mirrored to the other, and a backup tar file 
for the contents of the original database server is 
archived once per week. For linux users, one can 
start up own database server in her/his PC, ac- 
cording to a Belle instruction. User, if necessary, 
can download original database contents from a 
KEK B computer for private purpose. 

3. Belle Computing System 

3.1. Computing Model Overview 

Figure ^ indicates an overview of our comput- 
ing model for the Belle experiment. As can be 
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Figure 1. Computing model overview. 



seen, it comprises of the four major elements. 
The first one is the KEK B Computers which has 
been operated since February 20010. This has 
been a principal system, where data processing 
as well as analyses have been performed. In addi- 
tion to this, we have equipped a 60 TB disk stor- 
age area March 2003. In this space, more beam 
and MC data can be kept. As for a network in- 
side Japan, Super-SINET link has been available 
since 2002'?'. To enrich CPU's for user analyses, 
PC farms dedicated to analysis jobs have been 
installed in June 2003. These components are in- 
terlinked each other with a fast network of 1^10 
Gbps. 

3.2. KEK B Computers 

The KEK B computers are schematically 
shown in Figure [21 This system consists of the 
three cardinal links. The first one is called "com- 
puting network" , which interconnects the 38 Sun 
servers with the PC farms, which will be de- 
scribed below. These Sun hosts are operated as 
a batch job system controlled by the LSF sched- 
uler. The computing network is also attached to 
the DTF2 tape robotic device with 500 TB total 
volume which is used for rawdata and DST data 
storage. The next network links the 9 work group 
servers of the Sun hosts to the storage devices, 
which consists of the 8 TB file server and the 
hierarchy mass storage( HSM ) of 120 TB capac- 
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Figure 2. Belle computing system. 



ity with the 4 TB staging disk. The work group 
servers are connected via the "user network" to 
100 PC's for users. With this network, user can 
login from his/her PC to one of the work group 
servers, from which one can submit a batch job to 
the 38 Sun compute servers. Table ^ summarizes 
the CPU's allocated in the KEK B computers. 

3.3. PC farm 

More and more data have been accumulated, 
more and more computing needs for event pro- 
cessing have been seriously arising. To fulfill this 
requirements, a bunch of PC's has been installed 
and connected into this existing network by con- 
sidering the best usage without making a bottle- 
neck. TableEltabulates the PC farm CPU's in our 
system. As one can see, our PC farms are hetero- 
geneous from various vendors. The best choice at 
the moment to get new PC's usually made us to 



purchase from a variety of vendors, and it effec- 
tively reduces cost for CPU's. 

In all PC's, linux utilities as well as the Belle 
library packages have been loaded and we can use 
them as clusters of seamless PC systems to pro- 
cess event data. 

The primary purpose to add PC farms is that 
we have to increase CPU power for DST and MC 
productions. In 2003, we expanded usage for our 
PC's by releasing new PC farms for user analyses, 
as a possible solution for ever increasing users' 
demand. The 10 login servers have been arranged 
with 6 TB local disk area, where user histogram 
files and analysis codes are located. From each 
login server, job can be submitted to the user PC 
farm consisting of 84 PC's with dual Xeon 2.8 
GHz CPU's. All PC's are managed by the LSF 
queuing utility. From user PC farms, beam and 
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Table 1 



A summary of CPU's for the KEK B computers. 


host 


processor 


clock #CPU's 


#nodes 


computing 


server 


spare 500MHz 


4 


38 


work group serve 


spare 500MHz 


4 


9 


user PC 


P3 


IGHz 


1 


100 


Table 2 












A breakdown of the PC farm CPU's. 








vendor 


processor 


clock 


#CPU's 


#nodes total CPU 


Dell 


P3 


500MHz 


4 


16 


32GHz 


Dell 


P3 


550MHz 


4 


20 


44GHz 


Compaq 


P3 


800MHz 


2 


20 


32GHz 


Compaq 


P3 


933MHz 


2 


20 


37GHz 


Compaq 


Intel Xeon 


700MHz 


4 


60 


168GHz 


Fujitsu 


P3 


1.26GHz 


2 


127 


320GHz 


Compaq 


P3 


700MHz 


1 


40 


28GHz 


Appro 


Athlon 


1.67GHz 


2 


113 


377GHz 


NEC 


Intel Xeon 


2.8GHz 


2 


84 


470GHz 


Total 








500 


1508GHz 



MC data samples, which are stored in the 60 TB 
disk mentioned previously, can be accessed. 

3.4. Super-SINET at Belle 

A fast 1 Gbps network dedicated to academic 
activities, Super-SINET |7j, has been available 
between KEK and major Japanese universities. 
This link enables us to copy a bulk of beam and 
MC data from KEK to other institutions and 
vice versa. Moreover, computing resources can 
be shared as seamless system using Super-SINET. 
For example, one disk connected to PC at Nagoya 
university can be mounted to the KEK computer 
via NFS as if it were located inside the KEK site. 
Then, output data can be written onto the disk 
directly from the KEK computer. It is possible 
to make efficient and full use of total resources 
collaboration- wide. 

3.5. Data Management 

A large volume of data is comprised of more 
than 30 K data files including beam and MC 
events. Basically each user has to go through 
all of these files to obtain final physics results. 
So, it is very important to notify all of file loca- 
tions to users for their analyses. These data files 
are distributed over a bunch of storage disks and 



sometimes data files have to be moved for various 
reasons such as disk failure and so on. From ad- 
ministrative point of view, it is necessary to main- 
tain flexibility in data management despite of any 
change of the file locations. To solve this, we have 
registered all of the attributes of data files such 
as locations, data type and number of events into 
our database, and they serves as "metadata" to 
access actual beam data. The central database 
contents for data file information is maintained 
and updated whenever new data is available. The 
web-based interface between database and users 
is prepared to extract necessary information. In 
user's batch job, inquiry to the database is auto- 
matically issued and user can easily analyze event 
data without knowing actual file locations. 

4. Data Processing 

4.1. Beam data production 

The scheme of the DST production is shown in 
FigureOl In the first step of the DST production, 
one of Sun compute host is assigned as a tape 
server, and two DTF2 tapes, one is for raw data 
and the other for DST data, are mounted. Then, 
raw data are read from the tape and are dis- 
tributed over PC nodes. In each PC node, event 
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Figure 3. A schematic drawing of the DST production scheme. 



processing is performed in the B.A.S.F. frame- 
work. After the reconstruction is made, event 
data are sent back to the tape server, where data 
are written onto the tape as DST. 

The next step, the event skimming, is carried 
out in such a way that DST data are again read 
from the DST2 tape at the Sun compute server, 
and an appropriate selection criteria is imposed 
onto the data. In case that an event satisfies selec- 
tion conditions, it is saved onto disk as a skimmed 
event. Each event is examined by a set of selec- 
tions such as //-pair event for a detector study 
and hadronic event for a physics analysis. 

In our system, we can process about 1 fb~^ 
of beam data per day with 40 PC nodes of quad 
Intel Xeon 700 MHz CPU's each, and it can be in- 
creased up to 2 fb~^ per day by adding PC nodes. 
Figure 0] shows a history of our beam data pro- 
cessing from 2001. In 2001, we completed the first 



entire reprocessing with a single version of the re- 
construction package, providing 30 fb^^ of data 
sample for 2000 summer conferences. In 2002, 
major update of the software was made in April 
2002 and the second reprocessing for 78 fb^^ of 
data using this library was performed from April 
to June. Last year, we added another 80 fb~^ 
by reprocessing them, amounting to 159 fb^^ of 
total beam data. 

4.2. MC production 

We have three types of MC samples, B^B^^ 
B^ B~ and continuum events. They are produced 
on a run- by-run basis, where each MC sample file 
corresponds to each beam data file. After data 
production for one run beam file is done, run de- 
pendent information like beam interaction pro- 
file and background hit rate are calculated imme- 
diately and, based upon these information, MC 
sample data corresponding to the beam run file 
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Figure 4. History of reprocessing from 2001. 



Figure 5. MC production done in 2002 and 2003. 



is created for three types. Beam background hits 
are overlaied with MC generated data to mimic 
actual events as precisely as possible. Figure El 
shows how we have produced MC samples recent 
two years. We have produced 2.2 billion events 
in total by the end of 2003, which is equivalent 
to 3 times larger statistics for 159 fb~^ real beam 
data. 

Another aspect of the MC production is contri- 
bution from the remote institutes outside KEK. 
Approximately one quater of MC events have 
been produced at remote sites and they are trans- 
ferred to KEK via network. 

5. Summary and Plan 

The Belle computing system has been success- 
fully working and we have processed a bulk of 
beam data of more than 250 fb^^ so far. For 
MC data, 2.2 billion events have been produced, 
which corresponds to more than 3 times larger 
statistics than real data. Those data have been 
managed via our database and this scheme allows 
us to provide the data sets steadily and flexibly. 

In 2003 summer, new silion vertex detector has 



been installed it is expected that this detector ex- 
pands our tracking and vertexing capability fur- 
ther. At present, reconstruction algorithm based 
upon new Belle configuration is being developed. 
After it is completed, beam data in 2003 autumn 
runs will be processed. To do this, we are plan- 
ninng to double our processing power for beam 
data by adding more CPU's and storage devices. 



REFERENCES 



Nucl. 



Belle Collaboration, A. Abashian et al. 
Instr. and Meth. A479, 117(2002). 
S. Kurokawa and E. Kikutani, Nucl. Instr. 
and Meth. A499, 1(2003). 
Belle Collaboration, K. Abe et al. Phys. Rev. 
Lett. 87, 091802(2001). 

R. Brun et al, Geant3.21 CERN Report 

No.DD/E E/84-l(1987). 

[http : / Tw ww . postgresq l . coin7| 

I. Adachi et al, proc. of CIIEP03, La Jolla, 

Cahfornia, March 24-28, 2003. 

National Institute of Informatics, Japan, 

(http : / / www ■ sinet . ad . jp/| 



